A Comparison of Hybrid Incremental Reuse Strategies for Reinforcement Learning in Genetic Programming

Easy missions is an approach to machine learning that seeks to synthesize solutions for complex tasks from those for simpler ones. ISLES (Incrementally Staged Learning from Easier Subtasks) [1] is a genetic programming (GP) technique that achieves this by using identified goals and fitness functions for subproblems of the overall problem. Solutions evolved for these subproblems are then reused to speed up learning, either as automatically defined functions (ADF) or by seeding a new GP population. Previous positive results using both approaches for learning in multi-agent systems (MAS) showed that incremental reuse using easy missions achieves comparable or better overall fitness than single-layered GP. A key unresolved issue dealt with hybrid reuse using ADF with easy missions. Results in the keep-away soccer (KAS) [2] domain (a test bed for MAS learning) were also inconclusive on whether compactness-inducing reuse helped or hurt overall agent performance. In this paper, we compare reuse using single-layered (with and without ADF) GP and easy missions GPs to two new types of GP learning systems with incremental reuse. In our research we performed six experiments. The first experiment used standard, conventional GP without any enhancement. We will refer to this as single-layered GP. The second used using standard GP enhanced with ADF. We will refer to this as single-layered ADF. The rest of the experiments used double-layered (two stages of evolution). The third used ISLES with Standard GP in the first and second stage. We will refer to this as ISLES SGP/SGP. The fourth used ISLES with Standard GP in the first stage and ADF in the second stage. We will refer to this as ISLES SGP/ADF. The fifth used ISLES with ADF in the first stage and Standard GP in the second stage. We will refer to this as ISLES ADF/SGP. The sixth and final experiment used ISLES with ADF in the first and second stage. We will refer to this as ISLES ADF/ADF. Each experiment was done using ECJ [3] and a KAS simulator created by S. Gustafson [1]. For both single-layered experiments, the target concept was to minimize the number of turn overs. For all of the experiments with ISLES, the first stage goal was to maximize the number of successful passes between two teammates in the absence of takers. The second stage goal was to minimize the number of turnovers from keepers (3 keepers) to takers (1 taker). We took the average of ten runs for each experiment. The population size for all the experiments was 4000. For the single-layered experiments, we stopped at